Skip to content

support min_distance for large limit query - #2408

Open
Ningsir wants to merge 2 commits into
antgroup:mainfrom
Ningsir:main-min-distance
Open

support min_distance for large limit query#2408
Ningsir wants to merge 2 commits into
antgroup:mainfrom
Ningsir:main-min-distance

Conversation

@Ningsir

@Ningsir Ningsir commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Change Type

  • Bug fix
  • New feature
  • Improvement/Refactor
  • Documentation
  • CI/Build/Infra

Linked Issue

What Changed

Test Evidence

  • make fmt
  • make lint
  • make test
  • make cov, run tests, and collect coverage
  • Other (describe below)

Test details:

<!-- Paste commands and key output here -->

Compatibility Impact

  • API/ABI compatibility:
  • Behavior changes:

Performance and Concurrency Impact

  • Performance impact:
  • Concurrency/thread-safety impact:

Documentation Impact

  • No docs update needed
  • Updated docs:
    • README.md
    • DEVELOPMENT.md
    • CONTRIBUTING.md
    • Other:

Risk and Rollback

  • Risk level:
  • Rollback plan:

Checklist

  • I have linked the relevant issue (required for kind/bug and kind/feature; see "Linked Issue" above)
  • I have added/updated tests for new behavior or bug fixes
  • I have considered API compatibility impact
  • I have updated docs if behavior/workflow changed
  • My commit messages follow project conventions (Conventional Commits, optional [skip ci] prefix)

@mergify

mergify Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 🙋 you

Protection Waiting on
🔴 Require kind label 🙋 you
🟢 Require version label

🔴 Require kind label

Waiting for

  • label~=^kind/
This rule is failing.
  • label~=^kind/

Show 1 satisfied protection

🟢 Require version label

  • label~=^version/

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

@Ningsir
Ningsir force-pushed the main-min-distance branch 2 times, most recently from 6316d9b to 54fb3a0 Compare July 3, 2026 10:54
@LHT129 LHT129 self-assigned this Jul 8, 2026
@vsag-bot

vsag-bot commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao

Assisted-by: Cursor:claude-sonnet-4.6
Signed-off-by: xin ning <xinning@U-GJ4YX14D-0038.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: xin ning <xinning@U-GJ4YX14D-0038.local>
Signed-off-by: xin ning <xinning@U-GJ4YX14D-0038.local>
@Ningsir
Ningsir force-pushed the main-min-distance branch 2 times, most recently from d5c57bc to c3a308c Compare July 16, 2026 04:03
iter_ctx->PopDiscard();
}
} else {
lower_bound = std::numeric_limits<float>::max();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] When iter_ctx != nullptr && !iter_ctx->IsFirstUsed(), the discard nodes from the previous iteration are replayed into top_candidates without checking min_distance. This is inconsistent with the entry-point path (the else branch) and the neighbor-visit path, both of which apply the min_distance filter. If a discard node has a distance <= min_distance, it should not be added to top_candidates.

Consider adding a min_distance check here:

if (iter_ctx->CheckPoint(cur_inner_id) && cur_dist > min_distance + vsag::THRESHOLD_ERROR) {
    top_candidates.emplace(cur_dist, cur_inner_id);
    ...
}

top_candidates.emplace(dist, ep_id);
candidate_set.emplace(-dist, ep_id);
if (dist > min_distance + vsag::THRESHOLD_ERROR) {
top_candidates.emplace(dist, ep_id);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] In the else branch, when the entry point is valid but its distance is <= min_distance + THRESHOLD_ERROR, lower_bound remains std::numeric_limits<float>::max() (set at line 566). This means the search termination condition (-current_node_pair.first) > lower_bound will never trigger until top_candidates reaches ef size, potentially causing the search to explore more nodes than necessary.

In the original code, lower_bound was unconditionally set to dist when the entry point was valid. Consider whether lower_bound should still be set to dist even when the result is filtered out by min_distance, since lower_bound is used for search pruning, not result filtering.

if (dist <= radius + vsag::THRESHOLD_ERROR)
if (dist <= radius + vsag::THRESHOLD_ERROR && dist > min_distance + vsag::THRESHOLD_ERROR)
top_candidates.emplace(dist, ep_id);
candidate_set.emplace(-dist, ep_id);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] In the searchBaseLayerST range-search overload (the second template), the entry point check at the top of the function also uses min_distance for filtering top_candidates but does not adjust lower_bound when the entry point is filtered out. The same lower_bound concern applies here — lower_bound is set to dist regardless, which is correct for this overload since lower_bound is set before the min_distance check. However, the else branch (invalid entry point) sets lower_bound = std::numeric_limits<float>::max() which is fine.

This is just a note for consistency — the first overload (with iter_ctx) should follow a similar pattern where lower_bound reflects the actual search frontier, not the filtered frontier.

// Sign convention: top_candidates stores positive distances (nearest = smallest);
// candidate_set is a max-heap, so distances are negated (nearest = largest, popped first).
top_candidates->Push(cur_dist, cur_inner_id);
if (cur_dist > inner_search_param.min_distance + THRESHOLD_ERROR) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] In the basic_searcher.cpp iterator overload search_impl, when replaying discard nodes from iter_ctx (the !iter_ctx->IsFirstUsed() path), the min_distance check is applied to top_candidates->Push but NOT to candidate_set->Push. The candidate_set always gets the node pushed regardless of min_distance. This is correct for graph traversal — candidate_set should include all nodes for neighborhood expansion. However, the hnswalg.cpp searchBaseLayerST has the same pattern (candidate_set always gets the entry point), so this is consistent.

No action needed, just confirming the pattern is intentional.

vsag::IteratorFilterContext* iter_ctx = nullptr,
bool is_last_filter = false) const = 0;
bool is_last_filter = false,
float min_distance = std::numeric_limits<float>::lowest()) const = 0;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note] The min_distance parameter is plumbed through the searchKnn virtual interface in algorithm_interface.h with a default value of std::numeric_limits<float>::lowest(). This is backward-compatible for existing callers. However, there is no corresponding min_distance parameter added to searchRange in the same interface. If range search also needs min_distance support in the future, it would need a separate change.

Also, the bruteForce method does not receive min_distance. If brute-force fallback is used (e.g., via brute_force_threshold in HGraph), results below min_distance will not be filtered. Consider whether this is intentional or if bruteForce should also respect min_distance.

bool consider_duplicate{false};

// skip results with dist <= min_distance (for search iterator)
float min_distance{std::numeric_limits<float>::lowest()};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note] The min_distance parameter defaults to std::numeric_limits<float>::lowest() (approximately -3.4e38). The check dist > min_distance + THRESHOLD_ERROR will therefore always pass when min_distance is at its default value, since THRESHOLD_ERROR is 2e-6 and the sum is still effectively -3.4e38. This means the default behavior is a no-op, which is correct.

However, if a user sets min_distance to a very large positive value (e.g., 1e38), min_distance + THRESHOLD_ERROR could overflow to +inf due to floating-point precision limits, causing the check to always fail and returning zero results. This is an edge case, but worth being aware of.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR! The min_distance feature for filtering out results below a distance threshold in large-limit queries is a useful addition. Here is a summary of the review:

Overall assessment: The implementation is well-structured and follows the existing code patterns. The parameter is plumbed consistently through the search stack (HGraph → HNSW → BasicSearcher/ParallelSearcher). The default value of std::numeric_limits<float>::lowest() ensures backward compatibility.

Key concerns:

  1. [suggestion] iter_ctx replay path in hnswalg.cpp searchBaseLayerST: When replaying discard nodes from a previous iteration, min_distance is not checked. This is inconsistent with the entry-point and neighbor-visit paths.

  2. [suggestion] lower_bound in hnswalg.cpp searchBaseLayerST: When the entry point is valid but filtered out by min_distance, lower_bound remains at float::max(), which may cause the search to explore more nodes than necessary. Consider setting lower_bound to dist regardless for pruning purposes.

  3. [note] Brute-force fallback: The bruteForce method and searchRange interface do not receive min_distance. If these code paths are exercised with min_distance set, results will not be filtered.

  4. [note] Edge case: Very large min_distance values could cause floating-point overflow in the min_distance + THRESHOLD_ERROR check.

Overall the changes look correct and the approach is sound. The suggestions above are non-blocking improvements for consistency and edge-case handling.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The entry point in parallel_searcher.cpp (around line 179) is missing the min_distance check. In basic_searcher.cpp, the entry point push to top_candidates is guarded by dist > inner_search_param.min_distance + THRESHOLD_ERROR, but in parallel_searcher.cpp the check is absent:

// parallel_searcher.cpp, entry point handling
if (check_func(ep)) {
    top_candidates->Push(dist, ep);  // no min_distance check
    lower_bound = top_candidates->Top().first;
}

This means when parallel_search_thread_count > 1, the entry point will always be added to top_candidates regardless of min_distance, which is inconsistent with the single-threaded path in basic_searcher.cpp. Consider adding the same guard:

if (check_func(ep) && dist > inner_search_param.min_distance + THRESHOLD_ERROR) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review for PR #2408: support min_distance for large limit query

Issues Found

1. [suggestion] parallel_searcher.cpp:179 — Entry point missing min_distance check

The entry point in parallel_searcher.cpp is pushed to top_candidates without checking min_distance:

if (check_func(ep)) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

In contrast, basic_searcher.cpp applies the min_distance filter at its entry point (line 210 in the new code):

if ((not is_id_allowed || is_id_allowed->CheckValid(ep)) and
    dist > inner_search_param.min_distance + THRESHOLD_ERROR) {

This means the parallel searcher may return results with dist <= min_distance when the entry point itself is the best candidate.

2. [suggestion] hnswalg.cpp:1868-1872searchRange() does not pass min_distance to searchBaseLayerST

The searchRange public method calls searchBaseLayerST without passing min_distance:

top_candidates = searchBaseLayerST<false, true>(currObj, query_data, radius, ef, is_id_allowed);

This means range search completely ignores the min_distance parameter. If min_distance filtering is intended for range search as well (the range-search overload of searchBaseLayerST already accepts the parameter), searchRange should pass it through.

3. [note] hnswalg.cpp:690 — Range search lower_bound set to dist even when entry point filtered by min_distance

In the range-search overload of searchBaseLayerST, lower_bound is unconditionally set to dist before the min_distance check:

lower_bound = dist;
if (dist <= radius + THRESHOLD_ERROR && dist > min_distance + THRESHOLD_ERROR)
    top_candidates.emplace(dist, ep_id);

This is actually correct behavior for this overload — lower_bound reflects the actual search frontier distance, not the filtered frontier. This is consistent with the range-search semantics. However, it differs from the first overload (with iter_ctx) where lower_bound is only set when the entry point passes the min_distance filter, which was already noted by another reviewer.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review for PR #2408: support min_distance for large limit query.

This is a follow-up review. LHT129 already posted 6 comments on 2026-07-28 covering several issues that remain unresolved. I found one additional issue below.

New finding: ParallelSearcher entry point missing min_distance check

In src/impl/searcher/parallel_searcher.cpp line 179-181, the entry point is added to top_candidates without checking min_distance:

if (check_func(ep)) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

Compare with BasicSearcher::search_impl (basic_searcher.cpp line 210-211) which correctly applies the filter:

if ((not is_id_allowed || is_id_allowed->CheckValid(ep)) and
    dist > inner_search_param.min_distance + THRESHOLD_ERROR) {

This inconsistency means the parallel search path will return results below min_distance when the entry point happens to be close to the query. The fix is to add the same min_distance check to the parallel searcher entry point.

Previously reported issues (from LHT129, still unresolved):

  1. hnswalg.cpp:558-559iter_ctx discard node replay does not check min_distance
  2. hnswalg.cpp:566-573lower_bound stays at max() when entry point is filtered by min_distance, potentially causing excessive search exploration
  3. algorithm_interface.h:63bruteForce and searchRange methods do not receive min_distance parameter

}
if (not is_id_allowed || is_id_allowed->CheckValid(ep)) {
if ((not is_id_allowed || is_id_allowed->CheckValid(ep)) and
dist > inner_search_param.min_distance + THRESHOLD_ERROR) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] When the entry point is valid (passes check_func) but its distance is <= min_distance + THRESHOLD_ERROR, the entry point is filtered out of top_candidates and lower_bound remains at its initial value std::numeric_limits<float>::max(). This means the search termination condition (-current_node_pair.first) > lower_bound will never trigger until top_candidates reaches ef size, potentially causing the search to explore more nodes than necessary.

In the original code, lower_bound was unconditionally set to dist when the entry point was valid. Consider whether lower_bound should still be set to dist even when the result is filtered out by min_distance, since lower_bound is used for search pruning, not result filtering.

This is the same concern as in hnswalg.cpp line 572 (previously flagged).

bool consider_duplicate{false};

// skip results with dist <= min_distance (for search iterator)
float min_distance{std::numeric_limits<float>::lowest()};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[note] This PR adds a new min_distance search parameter across 12 files but does not include any test changes. The feature filters search results based on a distance threshold, which is a correctness-sensitive operation. Consider adding tests that verify:

  1. Results with distance <= min_distance are correctly excluded from search output
  2. The default value (std::numeric_limits<float>::lowest()) is a no-op (all results returned)
  3. Edge cases: min_distance set to a very large value returns empty results
  4. Interaction with filters and iterator-based search
  5. Interaction with range search (where both radius and min_distance constraints apply)

Since min_distance is plumbed through both the HNSW native path and the HGraph searcher path (basic_searcher + parallel_searcher), both paths should be tested.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The is_last_filter path in searchKnn (around line 1716-1722 in hnswalg.cpp) is missing a min_distance check.

When iter_ctx != nullptr && !iter_ctx->IsFirstUsed() && is_last_filter, the function directly dumps all discard nodes from iter_ctx into the result without filtering by min_distance:

if (is_last_filter) {
    while (!iter_ctx->Empty()) {
        uint32_t cur_inner_id = iter_ctx->GetTopID();
        float cur_dist = iter_ctx->GetTopDist();
        result.emplace(cur_dist, getExternalLabel(cur_inner_id));  // no min_distance check
        iter_ctx->PopDiscard();
    }
    return result;
}

This is inconsistent with every other code path that applies the min_distance filter (entry point, neighbor visit in searchBaseLayerST, and both basic_searcher/parallel_searcher paths). Consider adding:

if (cur_dist > min_distance + vsag::THRESHOLD_ERROR) {
    result.emplace(cur_dist, getExternalLabel(cur_inner_id));
}

This ensures the min_distance filter is consistently applied across all search paths, including the final iteration of a multi-step filtered search.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reviewed this PR in addition to the existing 8 review comments from LHT129. Here is one additional finding:

[suggestion] parallel_searcher.cpp entry point missing min_distance filter

The entry point handling in parallel_searcher.cpp (around line 183 in the search_impl with label_table) does not apply the min_distance filter, unlike basic_searcher.cpp (label_table overload at line 384) which was updated to check dist > inner_search_param.min_distance + THRESHOLD_ERROR. This means in the parallel search path, the entry point will always be added to top_candidates regardless of min_distance, creating an inconsistency between the two searcher implementations.

Current code in parallel_searcher.cpp:

if (check_func(ep)) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

Suggested fix:

if (check_func(ep) && dist > inner_search_param.min_distance + THRESHOLD_ERROR) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

This is the same pattern already applied in basic_searcher.cpp and should be consistently applied across both searcher implementations.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Found two additional locations where the min_distance filter is missing:

1. src/impl/searcher/parallel_searcher.cpp line 179 — entry point missing min_distance check

The entry point push to top_candidates at line 179-180 is missing the min_distance check. This is inconsistent with the same pattern in basic_searcher.cpp (line 387-388), which correctly applies dist > inner_search_param.min_distance + THRESHOLD_ERROR before pushing the entry point.

When the entry point distance is <= min_distance + THRESHOLD_ERROR, it should not be added to top_candidates. Additionally, lower_bound should still be set to dist for search pruning purposes (same concern as in hnswalg.cpp line 572 and basic_searcher.cpp line 211).

Suggested fix:

if (check_func(ep) && dist > inner_search_param.min_distance + THRESHOLD_ERROR) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

2. src/algorithm/hnswlib/hnswalg.cpp line 1715-1722 — is_last_filter fast path missing min_distance check

In the is_last_filter fast path of searchKnn, discard nodes from iter_ctx are directly emplaced into result without applying the min_distance filter. This means when is_last_filter is true, results with distance <= min_distance will incorrectly appear in the final output.

The is_last_filter path is used when the last filter in a chain is reached, and results are returned directly without further graph search. The min_distance filter should be applied here consistently with other paths.

Suggested fix:

if (is_last_filter) {
    while (!iter_ctx->Empty()) {
        uint32_t cur_inner_id = iter_ctx->GetTopID();
        float cur_dist = iter_ctx->GetTopDist();
        if (cur_dist > min_distance + vsag::THRESHOLD_ERROR) {
            result.emplace(cur_dist, getExternalLabel(cur_inner_id));
        }
        iter_ctx->PopDiscard();
    }
    return result;
}

// Sign convention: top_candidates stores positive distances (nearest = smallest);
// candidate_set is a max-heap, so distances are negated (nearest = largest, popped first).
top_candidates->Push(cur_dist, cur_inner_id);
if (cur_dist > inner_search_param.min_distance + THRESHOLD_ERROR) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] In the basic_searcher.cpp iterator overload of search_impl (the first template, around line 168), when replaying discard nodes from iter_ctx, if all discard nodes are filtered out by min_distance (i.e., cur_dist <= inner_search_param.min_distance + THRESHOLD_ERROR), top_candidates remains empty and lower_bound stays at its initial value std::numeric_limits<float>::max(). This causes the search termination condition (-current_node_pair.first) > lower_bound to never trigger until top_candidates reaches ef size, leading to unnecessary graph exploration.

This is the same class of issue as the lower_bound concern previously flagged in hnswalg.cpp:572 and basic_searcher.cpp:211lower_bound is used for search pruning, not result filtering, so it should reflect the actual search frontier regardless of min_distance filtering.

Consider setting lower_bound based on the discard nodes' distances even when they are filtered out of top_candidates, or alternatively, track a separate search_lower_bound for pruning purposes.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Review Notes

[suggestion] Missing min_distance check in HGraph is_last_filter path

In src/algorithm/hgraph/hgraph_search.cpp, the is_last_filter fast path in HGraph::KnnSearch (around lines 154-160 in the original code) pushes all discard nodes from iter_filter_ctx directly into search_result without filtering by min_distance:

if (is_last_filter) {
    while (!iter_filter_ctx->Empty()) {
        uint32_t cur_inner_id = iter_filter_ctx->GetTopID();
        float cur_dist = iter_filter_ctx->GetTopDist();
        search_result->Push(cur_dist, cur_inner_id);  // no min_distance check
        iter_filter_ctx->PopDiscard();
    }
}

This is the same issue previously flagged for hnswalg.cpp's searchKnn is_last_filter path. Both the HGraph and HNSW is_last_filter paths should consistently apply the min_distance filter:

if (cur_dist > params.min_distance + THRESHOLD_ERROR) {
    search_result->Push(cur_dist, cur_inner_id);
}

@@ -323,13 +323,15 @@ ParallelSearcher::search_impl(const GraphInterfacePtr& graph,
if (top_candidates->Size() < ef || lower_bound > dist ||

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The entry point in parallel_searcher.cpp is pushed to top_candidates without checking min_distance. This is inconsistent with basic_searcher.cpp (both templates, lines 210-211 and 387) where the entry point is guarded by dist > inner_search_param.min_distance + THRESHOLD_ERROR.

Current code (line 179-182):

if (check_func(ep)) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

Suggested fix:

if (check_func(ep) && dist > inner_search_param.min_distance + THRESHOLD_ERROR) {
    top_candidates->Push(dist, ep);
    lower_bound = top_candidates->Top().first;
}

Without this check, the parallel searcher path will include results with distance <= min_distance, while the basic searcher path correctly filters them out. This leads to inconsistent behavior depending on whether parallel search is enabled.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is a well-structured PR that adds min_distance filtering support across the search pipeline. The parameter is plumbed consistently through HNSW, HGraph, and the searcher layers, with sensible defaults that make it a no-op when not configured.

The existing inline comments have covered the key concerns:

  1. lower_bound pruning: When the entry point is filtered by min_distance, lower_bound remains at std::numeric_limits<float>::max(), causing unnecessary graph exploration. This affects hnswalg.cpp, basic_searcher.cpp, and parallel_searcher.cpp.
  2. Missing min_distance in bruteForce/searchRange paths: The brute-force fallback in HGraph and the searchRange interface do not receive min_distance.
  3. Missing tests: No test coverage for the new feature.
  4. is_last_filter paths: The discard-node replay in searchKnn and KnnSearch is_last_filter paths do not apply min_distance filtering.

The core implementation logic (filtering results by distance threshold) is correct. The main area for improvement is ensuring lower_bound reflects the actual search frontier for pruning purposes, independent of result filtering.

bool is_last_filter,
float min_distance) const {
std::shared_lock resize_lock(resize_mutex_);
std::priority_queue<std::pair<float, LabelType>> result;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The searchKnn method now accepts min_distance and passes it to searchBaseLayerST, but the is_last_filter fast-path (lines 1715-1721, unchanged in this diff) collects discard nodes from iter_ctx and returns them directly without applying the min_distance filter:

if (is_last_filter) {
    while (!iter_ctx->Empty()) {
        uint32_t cur_inner_id = iter_ctx->GetTopID();
        float cur_dist = iter_ctx->GetTopDist();
        result.emplace(cur_dist, getExternalLabel(cur_inner_id));  // no min_distance check
        iter_ctx->PopDiscard();
    }
    return result;
}

This means that in the last filter iteration of a multi-filter search, results with distance <= min_distance will be returned to the user, while all previous iterations correctly filter them out. Consider adding a min_distance check here:

if (cur_dist > min_distance + vsag::THRESHOLD_ERROR) {
    result.emplace(cur_dist, getExternalLabel(cur_inner_id));
}

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one additional issue not covered by the existing comments:

[suggestion] hgraph_search.cpp KnnSearch is_last_filter path lacks min_distance check

Similar to the issue flagged in hnswalg.cpp:1704, the is_last_filter fast-path in HGraph::KnnSearch (around the while (!iter_filter_ctx->Empty()) loop) does not apply min_distance filtering. When is_last_filter is true, all remaining candidates from iter_filter_ctx are pushed into search_result unconditionally, bypassing the min_distance threshold. Results with dist <= min_distance that should be skipped will still appear in the output.

The fix should add a min_distance check before search_result->Push() in this path, consistent with the fix needed in hnswalg.cpp.

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional review findings for the min_distance feature:

  1. hgraph_search.cpp is_last_filter path (line 125): The is_last_filter fast-path in HGraph::KnnSearch (iterator overload) collects discard nodes from iter_filter_ctx and pushes them directly into search_result without checking min_distance. This is the same issue as the is_last_filter path in hnswalg.cpp:1704 — results with distance <= min_distance will be returned to the user in the last filter iteration.

  2. hgraph_search.cpp brute_force_search (line 290): When brute_force_threshold triggers the brute-force fallback in SearchWithRequest, min_distance is not applied to filter results. The brute_force_search method does not accept or use min_distance, so results below the threshold will not be filtered when the brute-force path is taken. This is the HGraph counterpart of the bruteForce gap noted in the algorithm_interface.h comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feat](hnsw/hgraph): support large limit query

4 participants